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Assessment  of  a  Multi-University  Unmanned  Systems  Capstone 

Design  Project 


Abstract 

In  this  paper  we  discuss  the  assessment  methods  for  a  senior  capstone  design  project  involving 
teams  from  three  geographically  separated  universities ,  as  well  as  the  challenges  the  students 
faced  and  lessons  learned.  The  project  title  was  the  Joint  Cooperative  Unmanned  Systems 
Initiative  (JCUSI).  Each  team  was  tasked  with  developing  an  unmanned  autonomous  system 
operating  in  a  different  medium  (air,  water,  and  ground)  to  cooperatively  work  together  to 
complete  a  mission  of  protecting  a  harbor.  JCUSI  is  unique  in  that  the  customer  funding  the 
project  will  most  likely  employ  the  students  involved  either  as  engineers  implementing  future 
unmanned  systems  or  as  operators  of  unmanned  systems.  Consequently,  the  sponsor  was 
involved  in  defining  the  learning  outcomes  of  the  project,  which  were  added  to  our  normal 
pedagogical  outcomes  for  this  capstone  engineering  design  course. 

1 .  Introduction 

Multidisciplinary  senior  design  capstone  projects  have  been  popular  at  many  institutions  for 
several  years.  Multidisciplinary  projects  are  encouraged  by  the  Accreditation  Board  for 
Engineering  and  Technology’s  requirements  for  a  “realistic”  major  design  experience,1  with  the 
recognition  that  projects  in  industry  typically  require  multi-disciplinary  teams.  Another  recent 
capstone  trend  reflecting  life  in  industry  is  projects  with  geographically  separated  teams.  These 
teams  can  range  from  multi-university  teams  in  the  same  country2  to  international  multi¬ 
university  teams?'4  In  addition  to  the  traditional  challenges  of  multidisciplinary  teams,  these 
teams  are  also  faced  with  the  challenges  of  being  geographically  separated,  often  with  a  different 
language  or  culture.  Challenges  include  scheduling  difficulties  due  to  different  time  zones  and 
school  vacation  schedules;  coordination  and  communication  challenges,  not  only  due  to  not 
being  co-located,  but  also  due  to  different  languages;  and  the  impact  of  cultural  differences 
between  institutions,  leading  to  different  design  and  process  approaches.3,4  The  students  find 
that  defining  and  documenting  interfaces  becomes  even  more  important  when  geographically 
separated.3 

In  this  paper  we  will  look  at  an  example  of  a  project  with  teams  from  three  geographically 
separated  colleges:  The  U.S.  Air  Force  Academy  in  Colorado  Springs,  Colorado,  the  U.S. 
Military  Academy  in  West  Point,  New  York,  and  the  U.S.  Naval  Academy  in  Annapolis, 
Maryland.  Each  team  was  tasked  with  developing  an  unmanned  autonomous  system  operating  in 
a  different  medium  (air,  water,  and  ground)  to  cooperatively  work  together  to  complete  a  mission 
of  protecting  a  harbor.  This  project  is  unique  in  that  the  customer  funding  the  project  will  most 
likely  employ  the  students  involved  either  as  engineers  implementing  future  unmanned  systems 
or  as  operators  of  unmanned  systems.  The  sponsor  was  involved  in  defining  the  learning 
outcomes  of  the  project,  which  were  added  to  our  normal  pedagogical  outcomes  for  this  capstone 
engineering  design  course.  These  additional  outcomes  added  assessment  methods  to  our 
traditional  course  assessment.  Before  discussing  the  assessment  methods,  to  help  motivate  the 
problem  we  will  first  discuss  the  scenario  the  three  unmanned  systems  were  challenged  to  solve, 


the  engineering  challenges,  and  the  technical  results.  We  then  discuss  the  desired  outcomes,  the 
assessment  methods,  and  the  assessment  results, 

2.  Scenario 

JCUSI  is  an  undergraduate  research  project  composed  of  teams  from  three  universities  to  explore 
a  cooperative  control  scenario  involving  multiple  unmanned  aerial  vehicles  (UAVs),  unmanned 
surface  vehicles  (US Vs),  and  unmanned  ground  vehicles  (UGVs),  as  illustrated  in  Figure  1 . 

Each  team  designed  the  UxVs  their  respective  institution  specialized  in.  In  addition,  one  team 
designed  the  combined  command  center  (CCC).  These  unmanned  systems  (UxVs)  had  to 
cooperatively  and  autonomously  protect  a  harbor  from  intruding  boats.  The  scenario  begins  with 
two  UAVs  searching  the  harbor  entrance  attempting  to  identify  and  track  any  incoming  boats. 

To  simplify  the  task,  the  intruding  boat  has  a  unique  signature  (bright  orange  color).  Upon 
detecting  the  intruding  boat,  the  UAV  notifies  its  ground  station,  which  in  turn  sends  the  detected 
target  coordinates  to  the  combined  command  center,  which  tasks  a  USV  to  intercept  the 
intruding  boats.  The  other  UAV  continues  to  search  for  other  possible  intruding  boats,  while  the 
first  UAV  continually  tracks  the  detected  intruding  boat  and  sends  the  location  information  to  the 
USV  via  the  UAV  ground  station  and  the  combined  command  station.  When  the  USV  intercepts 
the  intruding  boat,  it  notifies  the  combined  command  center,  which  informs  the  UAVs,  and  then 
escorts  the  boat  to  the  shore.  At  this  point,  the  UAVs  must  detect  and  track  another  target,  a 
human  departing  the  boat  with  a  unique  signature  (bright  orange  color,  but  smaller  size).  The 
UAVs  loiter  above  the  boat  waiting  to  detect  and  track  the  human  target  leaving  the  boat.  When 
the  UAVs  detect  the  human  target,  they  notify  their  ground  station  that  sends  a  message  to  the 
combined  command  center,  which  then  tasks  the  UGVs  to  intercept  the  human.  The  scenario 
ends  when  the  UGVs  intercept  the  intruder  and  notify  the  combined  command  center. 


Figure  1.  Harbor  Protection  Scenario.  The  tanks  represent  the  UGVs,  the  green 
boats  represent  the  USVs,  the  airplanes  represent  the  UAVs,  and  the  orange  boat 
represents  the  intruder. 


3.  Engineering  Challenges  /  Technical  Results 


The  students  had  several  engineering  challenges.  First,  they  had  to  design  the  top-level 
cooperative  control  architecture  between  three  sets  of  unmanned  systems  as  well  as  the 
communication  infrastructure  and  protocols  to  support  the  architecture.  The  requirement  was  for 
the  system  to  work  fully  autonomously;  however,  human  intervention  was  allowed  for 
confirmation  of  targets  and  overriding  actions  taken  autonomously  by  the  UxVs.  Secondly,  the 
processing  and  fusion  of  the  heterogeneous  sensor  data  from  the  three  different  platforms  to  track 
the  targets  while  propagating  the  proper  error  ellipse  proved  to  be  very  difficult.  Thirdly,  the 
communication  architecture  had  to  support  the  bandwidth  of  multiple  video  streams  at  distances 
up  to  a  mile  range  with  limited  power  radios.  Finally,  the  architecture  had  to  ensure  all  the 
ground  stations,  UAVs,  USVs,  and  UGVs  maintained  “situational  awareness”  for  their  respective 
UxVs  in  a  timely  manner  for  control  and  safety  while  providing  an  overall  situational  awareness 
in  the  combined  command  center,  which  is  responsible  for  the  overall  mission.  The  combined 
command  center  is  shown  in 
Figure  2. 


Figure  2.  The  combined  command  center  (CCC),  showing  the  ground  control 
stations  for  the  UAVs,  UGVs,  and  USVs. 

Each  team  had  to  tackle  the  low-level  challenges  unique  for  their  platform.  For  example,  the 
UAV  team  had  to  design  or  integrate  the  following  subsystems:  (1)  an  onboard  computer  system; 
(2)  an  onboard  sensor  system;  (3)  an  autopilot  system;  (4)  a  ground  control  station,  Figure  3;  (5) 
a  communication  system  between  the  two  UAVs  and  the  UAV  ground  station,  which 
communicates  with  the  combined  command  center;  (6)  image  recognition  software  to  detect  and 
track  moving  boat  and  human  targets  (7)  a  backup  manual  radio  control  flight  control  system; 
and  (S)  onboard  power  for  propulsion  and  the  payload. 


Figure  3.  Example  of  the  Graphical  User  Interface  for  the  UAV  Ground  Control 
System.  The  left  images  are  live  images  from  the  two  UAVs  and  the  center 
picture  is  the  situational  awareness  image,  with  an  icon  representing  the  location 
of  one  UAV,  and  the  green  icon  representing  the  location  of  one  of  the  UGVs. 


Figure  4.  The  UAV,  USV,  and  UGV  used  in  the  project. 


Figure  4  illustrates  the  three  unmanned  vehicles.  In  the  final  demonstration,  each  team  was  able 
to  get  their  UxVs  to  work,  but  did  not  meet  all  the  technical  requirements.  The  autonomous 
tasks  achieved  are  listed  in  Table  1 ,  The  UAV  was  able  to  detect,  track,  and  relay  tracking 
information  of  the  target  boat  to  the  UAV  ground  station.  The  UAV  autonomously  detected  the 
candidate  target  and  a  man-in-the-loop  made  the  final  decision  to  confirm  the  target  at  the  UAV 
ground  station.  The  USV  was  capable  of  autonomous  navigation,  but  due  to  a  mechanical  failure 
was  unable  to  intercept  and  escort  the  intruder  boat  autonomously.  The  USV's  sensors  were  able 
to  identify  the  target.  The  smaller  size  and  glare  of  the  human’s  signature  prevented  auto¬ 
detection  by  the  UAVs,  so  a  man-in-the-loop  at  the  UAV  ground  station  had  to  provide  the 
human  target  location  to  the  UGVs.  The  UGVs  then  successfully  autonomously  intercepted  the 
ground  target.  The  biggest  technical  issue  was  a  failure  to  complete  the  final  integration  of  the 
overall  command  center  (CCC)  with  each  team’s  ground  control  station,  requiring  a  man-in-the- 
loop  to  rely  target  and  status  information  during  the  demonstration.  The  students  learned  the 
lesson  that  they  cannot  just  focus  on  getting  their  UxV  system  working,  but  have  to  worry  about 
the  details  of  interfacing  with  the  other  systems  and  not  taking  the  interfaces  for  granted.  Next 
year’s  JCUSI  project  hopes  to  successfully  apply  this  lesson  learned. 


Manual 

Autonomous 

UAV 

Take  Off  and  Landing 

Plan  Waypoints  based  on 
given  Search  Area 

Designate  Search  Area 

Fly  &  Navigate  to  Waypoints 

Confirm  Target  proposed  by 
UAV 

Detect  and  propose  Candidate 
Boat  Targets 

♦Detect  and  Track  Human 
Targets 

Track  Confirmed  Boat  Target 

Communicate  Target  /  Status 
to  UAV/CCC  ground  stations 

USV 

*  Intercept  Target 

Navigate  to  Estimated  Target 
Location 

♦Communicate  with 

CCC/UxV  stations 

Detect/Identify  Target 

UGV 

Navigate  to  Estimated  Target 
Location 

Detect  and  Intercept  Target 

Communicate  with  CCC/UGV 
stations 

Table  1.  Autonomous  versus  Manual  control  actually  achieved.  *  indicates  tasks 
intended  to  be  autonomous,  but  not  achieved. 

4.  Outcomes 

The  agreed  desired  outcomes  of  the  JCUSI  project  were  to  develop  young  engineers  who: 


(1 )  Understand  the  current  capabilities  and  limitations  of  unmanned  system  technology 

(2)  Can  identify  operational  opportunities  for  unmanned  systems 


(3)  Are  able  to  develop  and  articulate  unmanned  system  requirements 

(4)  Arc  able  to  function  as  part  of  a  multi-institute,  geographically  dispersed  team. 

While  outcomes  1  and  2  are  unmanned  systems  focused,  the  challenges  presented  by  outcome  3 
(requirements)  and  outcome  4  (geographically  separated  teams)  may  be  of  interest  to  many 
system  engineering  projects. 

In  addition  to  these  outcomes,  we  also  assessed  our  regular  capstone  design  course  outcomes, 
which  assess  the  students’  performance  following  a  defined  rubric  after  each  major  project 
milestone.  Our  project  milestones  are  the  system  requirements  review  (SRR),  the  preliminary 
design  review  (PDR),  the  critical  design  review  (CDR),  two  system  status  reviews  (SSR),  the 
system  verification  review  (SVR),  and  the  final  demonstration. 

5.  Assessment  Methods 

Three  formal  assessment  activities  were  defined  before  the  project  started: 

(1)  A  38-question  survey  taken  by  the  students  at  the  beginning  and  the  end  of  their 
participation  in  the  project  to  measure  their  perceptions  of  their  knowledge,  skills,  and 
attitudes  as  they  pertain  to  the  four  project  learning  outcomes. 

(2)  The  normal  course  assessment  tools  used  in  our  two- semester  capstone  engineering 
design  course  (grades  from  the  above  program  reviews) 

(3)  The  project  mentors’  qualitative  evaluation  of  the  team’s  achievements. 

In  addition,  other  assessment  opportunities  became  available  during  the  course. 

(1)  The  students  had  an  opportunity  to  visit  a  UAS  operational  site  mid-way  through  the 
project,  so  we  conducted  a  survey  to  access  the  impact  of  meeting  the  real  customers  in 
the  operational  environment  as  they  related  to  the  four  project  learning  outcomes. 

(2)  The  students  from  the  three  universities  had  an  opportunity  to  meet  in  person  mid- way 
through  the  project,  and  we  conducted  a  survey  to  access  the  impact  of  this  meeting  on 
the  four  project  learning  outcomes. 

(3)  Reflective  papers  by  the  students  of  their  experience  with  the  course, 

6.  Assessment  Results 
a.  Survey  Results 

Figure  5  presents  the  results  for  the  38-question  surveys  that  directly  measured  the  students’ 
perceptions  of  their  attainment  of  the  desired  project  outcomes.  Appendix  A  contains  the  survey 
questions.  All  of  students  on  the  team  took  both  surveys  and  each  answered  all  the  questions. 
The  chart  shows  the  average  response  to  each  question  at  the  beginning  of  the  project  and  at  the 
end.  The  questions  are  grouped  by  project  objective  with  the  first  objective’s  14  questions  the 
first  group  on  the  left. 

Of  note  in  Figure  5  is  that  the  students’  average  responses  for  each  objective  are  markedly  higher 
at  the  end  of  the  project.  The  largest  improvement  was  for  Objective  1 -‘understand  unmanned 


systems  capabilities  and  limitations’  where  their  average  response  changed  from  slightly  above 
“Disagree”  to  1/2  way  between  “Agree”  and  “Strongly  Agree.”  Figure  5  demonstrates  that  the 
students  felt  the  project  significantly  helped  them  toward  the  project  objective. 

For  Objective  4,  the  students  appear  to  believe  this  project  helped  them  with  experience  working 
on  geographically  separated  teams  with  different  cultures.  The  survey  showed  improvement  for 
all  questions,  such  as: 

“I  can  manage  deadlines  across  times  zones” 

“I  can  coordinate  meetings  and  teleconferences  across  time  zones  and  can  lead  and 
contribute  to  these  meetings  using  language  familiar  to  all  teams/communities” 

“I  can  better  understand  the  culture  and  expectations  of  the  other  teams/communities” 

“1  am  able  to  develop  visual  aids  to  communicate  concepts  to  those  not  physically 
present” 

“I  can  clarify  expectations  to  the  geographically  separated  teams” 

“I  have  a  better  appreciation  for  the  problems,  constraints,  and  solutions  that  the  different 
teams  encounter.” 


X 

C 

o 

ft 

a> 

C£ 

01 

tiO 


JCUSI  Survey  Results 
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Figure  5.  Students’  Perceptions  of  their  Attainment  of  the  JCUSI  Project  Objectives 

The  students  working  on  the  JCUSI  project  made  an  orientation  trip  to  an  unmanned  systems 
operational  site  early  in  the  course  to  view  training  operations  and  interact  with  pilots  and 
operators  flying  operational  Unmanned  Aircraft  Systems  (UAS)  missions.  The  impact  of  the  trip 
was  assessed  with  a  survey  given  before  and  after  the  trip.  Figure  6  displays  the  results  and 


Appendix  B  lists  the  questions.  Questions  13-16  were  added  for  the  survey  after  the  students 
returned  to  measure  specific  desired  learning  outcomes.  Of  interest  in  these  results  is  that  the 
students  had  high  expectations  for  the  trip  (Question  2)  and  the  trip  met  their  expectations.  The 
students  achieved  the  learning  outcomes  as  the  average  responses  ranged  from  “Strongly  Agree” 
to  “Somewhat  Agree”  after  the  trip  except  for  Question  6.  The  students  seemed  more  interested 
in  designing  UAS  than  being  the  user/operator  of  a  UAS. 
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Figure  6.  Results  of  the  Unmanned  Systems  Operational  Site  Trip  Survey 

The  final  survey  results  in  Figure  7  show  the  significant  increase  in  knowledge,  skills,  and 
positive  attitudes  after  the  students’  trip  to  meet  with  their  counterparts  at  the  other  universities 
midway  through  the  project.  Although  students’  anecdotal  comments  tended  to  decry  the  lack  of 
coordination  they  were  able  to  obtain  at  the  meeting  on  specific  technical  details,  it  is  very  clear 
that  the  cumulative  effects  of  the  unmanned  systems  operational  site  trip  and  the  trip  to  meet  the 
other  university  teams  have  had  a  very  significant  positive  impact. 


Survey  Results  after  Midway  Meeting  of  all  the  Teams 
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Figure  7.  Survey  Results  after  the  Meeting  of  all  Teams  Midway  through  the  Course 


b.  Assessment  using  the  Capstone  Course  Tools 

We  also  assessed  the  team  using  our  normal  course  assessments,  which  primarily  follow  our 
project  milestones.  The  project  milestones  are  the  system  requirements  review  (SRR),  the 
preliminary  design  review  (PDR),  the  critical  design  review  (CDR),  two  system  status  reviews 
(SSR),  the  system  verification  review  (SVR),  and  the  final  demonstration.  The  most  applicable 
for  the  assessment  of  our  customer  objective  #3  (requirements)  is  the  Systems  Requirements 
Review  (SRR).  We  will  not  present  our  students  “grades”  in  this  paper,  but  have  included  the 
detailed  “expectations/rubrics”  we  used  to  assess  the  students’  performance  in  the  SRR  in 
Attachment  3  as  an  example.  Contact  the  author  if  you  are  interested  in  the  detailed  rubrics  we 
use  for  the  other  reviews. 

c.  Project  Mentor’s  Assessment 

The  six  mentors  involved  with  this  project  met  after  the  end  of  the  semester.  The  following  are 
the  mentors’  conclusions  and  lessons  learned  grouped  by  the  JCUSI  objectives: 

1 .  Understand  the  current  capabilities  and  limitations  of  unmanned  system  technology 

•  The  trip  to  a  UAS  operational  site  was  a  great  motivator  and  learning 
experience.  Students  greatly  increased  their  understanding  of  the  problems 
inherent  in  autonomous  operation  and  automatic  target  detection. 

•  The  students  gained  an  appreciation  that  it  takes  sustained  effort  and 
persistence  to  successfully  coordinate  operations  and  control  of  three  different 
unmanned  systems  to  achieve  a  goal. 

2.  Can  identify  operational  opportunities  for  unmanned  systems 

•  One  team’s  passion  for  unmanned  systems  as  essential,  life-saving  systems 
made  a  big  impression  on  the  other  teams. 

•  Learned  enough  about  capabilities/limitations  to  make  better  decisions  on 
employing  unmanned  systems. 

3.  Are  able  to  develop  and  articulate  unmanned  system  requirements 

•  The  team  struggled  to  find  a  systematic  way  to  flush  out  all  the  requirements 
for  this  complex  scenario  and  system.  More  guidance  from  mentors  may  have 
helped. 

•  The  students  perhaps  learned  more  from  their  failure  to  identify  verifiable 
requirements  than  from  doing  it  properly  the  first  time. 

•  The  process  of  testing  each  requirement  fell  apart  at  the  end  when  the 
schedule  got  tight. 

4.  Are  able  to  function  as  part  of  a  multi-institute,  geographically  dispersed  team 

•  Enthusiasm  for  working  with  the  other  universities  significantly  increased 
after  trip  to  meet  the  other  teams. 

•  The  students  understand  much  better  the  difficulty  involved  in  coordinating 
and  conducting  meetings  working  across  time  zones  with  different  academic 
and  vacation  schedules. 

•  Greatly  appreciate  the  difficulty  in  achieving  true  interoperability  and  the 
importance  of  agreeing  to  and  maintaining  communication  interfaces. 


•  Logistically  and  operationally  were  much  better  prepared  than  the  previous 
year.  The  host  university  preparations  were  excellent  with  dedicated  work 
areas  for  each  team,  machine  shop  support,  and  a  prepared  landing  strip. 

•  Lessons  Learned:  Communications  interfaces  should  be  defined  early,  not  be 
changed  without  full  coordination,  and  be  tested  remotely  prior  to 
demonstration 

•  Lessons  Learned:  Teams  should  develop  a  coordinated,  detailed  Joint  Test 
Plan  for  the  prep  day  and  the  demonstration 

•  Lessons  Learned:  Should  hold  more  teleconference  or  video  conference 
coordination  meetings;  an  earlier  face-to-face  meeting  in  the  fall  would  help 
set  up  these  meetings. 

7.  Student  Reflections  and  Lessons  Learned 

The  students  each  wrote  an  essay  reflecting  on  the  insights  and  benefits  they  gained  and  lessons 
learned  from  the  project.  Many  of  the  comments  involved  challenges  working  with  geo¬ 
graphically  dispersed  teams  and  the  different  cultures  of  the  teams.  Every  team  was  eager  to 
work  with  the  others,  but  they  all  had  different  biases,  perceptions,  and  stereotypes  of  the  other 
teams.  This  tended  to  go  away  after  they  became  better  acquainted. 

The  different  cultures  created  challenges  even  with  simple  tasks.  Each  school,  had  developed 
their  own  terminology  for  their  unmanned  systems  (air,  water,  and  ground  systems  have  different 
terminology  for  the  same  functions),  which  was  an  obstacle  in  communicating  when  designing 
the  top-level  architecture.  For  example,  the  unmanned  aerial  system  team  liked  to  use  the  work 
“search”  for  the  same  task  the  unmanned  ground  system  team  used  for  “recon.”  This 
communication  problem  wasted  a  lot  of  time  in  the  early  teleconferences.  There  were  issues 
agreeing  on  data  formats.  For  example,  one  team  wanted  DMS  (degrees-minutes-seconds)  for 
coordinates,  while  another  insisted  on  MGRS  (Military  Grid  Reference  System).  They  also 
found  each  team  had  developed  their  own  approach  to  solving  the  problem  and  it  is  hard  to  get 
people  to  change  course  after  they  have  invested  time  and  effort  in  their  own  approach. 

Being  geographically  separated  brought  up  several  challenges.  Each  university  and  each  student 
had  different  schedules,  time  zones,  and  school  holidays  making  scheduling  teleconferences  with 
all  the  key  players  difficult.  Doing  the  integration  testing  of  the  systems  remotely  was  difficult, 
so  the  teams  put  off  much  of  the  integration  testing  until  they  would  be  co- located  the  day  before 
the  final  demo.  This  did  not  work  well,  as  all  the  teams  found  they  had  problems  with  their  own 
unmanned  systems  to  fix,  so  much  of  the  joint  integration  testing/debugging  was  not  done  until 
the  live  demo. 

There  were  also  many  comments  about  the  challenges  facing  unmanned  systems  in  the  operating 
environment,  their  limitations,  and  the  amount  of  autonomy  that  can  or  should  be  given  to  the 
systems.  They  learned  that  there  are  many  trade-offs  between  being  fully  autonomous  or  fully 
human  controlled,  and  that  the  best  systems  need  a  method  whereby  the  user  can  control  the 
level  of  autonomy,  taking  control  of  the  system  when  needed.  They  also  learned  about  Murphy’  s 
Law,  which  applies  especially  well  to  unmanned  systems.  They  gained  an  appreciation  for  not 
being  overly  optimistic  and  more  cautious  in  system  design  and  scheduling. 


Finally,  the  last  group  of  comments  surprised  us  in  that  our  students  gained  more  appreciation  for 
the  importance  management  and  system  engineering  has  in  these  complex  projects.  Our  students 
were  from  the  disciplinary  majors  of  electrical  engineering  and  computer  engineering,  and  came 
into  the  course  with  little  respect  for  system  engineering,  thinking  the  “real”  engineering  is  at  the 
disciplinary  level.  One  comment  was  “I  learned  a  lot  more  about  Systems  Engineering 
Management  and  the  need  to  think  about  every  possible  detail.”  One  of  the  biggest  benefits  they 
believe  they  have  gained  is  having  developed  much  stronger  planning  skills  for  complex 
systems. 

8.  Conclusions 

The  students  were  able  to  experience  the  management  and  engineering  challenges  of  a  large 
engineering  project  while  attempting  to  get  three  geographically  separated  teams  to  work 
together.  Coordination  difficulties  were  exacerbated  by  two  different  time  zones,  different  work 
schedules,  different  terminology,  and  varying  philosophies  on  how  to  solve  the  problem. 
Additionally,  each  team  faced  their  own  internal  challenges  with  their  individual  multi¬ 
disciplinary  teams.  In  the  end,  the  final  demo  was  mostly  a  success,  but  the  teams  learned  a 
valuable  lesson  about  the  importance  of  defining  interfaces  as  their  systems  all  had  trouble 
communicating  with  the  combined  command  center. 

We  found  the  customer-generated  objectives  useful  for  assessing  the  outcomes  for  this  capstone 
team.  We  initially  only  had  funds  for  the  teams  to  travel  to  the  final  demonstration,  but 
fortunately  the  sponsor  found  funds  for  two  other  trips  mid-way  during  the  design  phase,  which 
were  very  beneficial.  We  highly  recommended  for  other  similar  projects  to  (1)  visit  a  site  that 
uses  the  intended  systems  to  sec  the  challenges  and  limitations  faced  by  the  operators  and 
maintainers  of  the  systems,  and  (2)  meet  in-person  with  the  other  teams  as  early  as  possible  to 
help  establish  rapport  and  facilitate  later  remote  communication  among  the  teams. 
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Appendix  A.  JCUSI  Student  Survey 


Objective  1:  Understand  the  Capabilities  and  Limitations  of  Unmanned  Systems 

1 .  I  can  describe  how  common  sensors  in  unmanned  systems  are  typically  used  and  the  advantages  and 
disadvantages  they  offer. 

2 .  From  working  on  this  project,  I  have  a  clearer  understanding  of  the  capabilities  and  limitations  of  air,  ground, 
and  marine  vehicles. 

3,  I  understand  the  typical  signal  processing  that  an  unmanned  system  must  perform. 

4.  1  know  how  to  integrate  control  system  design  concepts  into  an  unmanned  system  design. 

5.  I  can  articulate  the  levels  of  autonomy  and  required  key  aspects  of  the  autonomy  algorithms 

6,  I  can  perform  platform  and  sensor  selection  using  objective  criteria. 

7.  I  know  how  to  develop  the  requirements  necessary  to  interface  different  platforms  and  subsystems. 

8.  1  can  plan  for  the  necessary  logistical  requirements  involved  in  testing  and  operating  an  autonomous  system. 

9.  I  have  the  general  skills  necessary  to  debug  and  troubleshoot  an  unmanned  system  in  the  field. 

10  I  have  an  appreciation  of  the  challenges  faced  by  field  operators  of  unmanned  systems. 

1 L 1  understand  the  level  of  robustness  and  redundancy  required  of  fielded  unmanned  systems. 

12. 1  understand  the  capability  gap  between  prototypes  and  systems  that  are  ready  to  be  deployed  and  fielded. 

13. 1  have  a  better  appreciation  for  the  challenges  in  developing  robust  and  fully  autonomous  solutions. 

14. 1  understand  why  unmanned  systems  are  important  to  DoD. 

Objective  2:  Identify  Operational  Opportunities  for  Unmanned  System  Solutions 

L  I  know  many  of  the  capabilities  and  limitations  of  unmanned  systems  and  can  determine  the  best  role  for  them 
in  the  operational  force. 

2.  I  can  list  many  of  the  challenges  unmanned  systems  face  in  the  operational  force. 

3.  I  can  identify  casks  that  can  he  potentially  automated  or  replaced  by  an  unmanned  system. 

4.  1  can  identify  the  level  of  autonomy  required  for  a  particular  task  and  can  determine  the  potential  role  of  a 
human  operator. 

5.  I  can  estimate  the  time  required  to  develop  a  system  and  the  probability  of  success  of  an  approach. 

6,  1  believe  that  autonomous  systems  have  the  potential  to  enhance  military  operations 

7.  I  feel  that  defense  industrial  partners  and  government  laboratories  are  equal  partners  in  developing  solutions  for 
unmanned  systems. 

Objective  3;  Develop  and  Articulate  Unmanned  System  Requirements  and  Specifications 

1.  1  have  knowledge  about  the  current  state-of-the-art  commercially  available  unmanned  systems. 

2.  1  understand  how  operational  needs  can  translate  to  the  technical  requirements  of  a  system. 

3.  I  can  use  a  formal  engineering  design  process  to  generate  the  specifications  and  performance  measures  from 
high  level  requirements. 

4.  I  can  separate  the  desired  functionality  from  a  specific  design  solution. 

5.  1  understand  the  importance  of  possessing  both  technical  and  operational  skills  to  generate  a  requirement. 

6.  I  appreciate  the  need  for  testable  or  demonstrable  requirements. 

7.  I  understand  that  vendors  and  non-military  personnel  often  use  a  different  terminology  and  have  a  different 
culture  than  the  military. 

Objective  4:  Function  as  Part  of  a  Multi -institute,  Geographically  Dispersed  Team- 

1.  I  better  understand  the  terminology  used  by  other  teams/communities. 

2.  I  better  understand  the  culture  and  expectations  of  the  other  teams/communities. 

3.  1  can  coordinate  meetings  and  teleconferences  across  time  zones  and  can  lead  and  contribute  to  these  meetings 
using  language  familiar  to  all  teams/communities. 

4.  I  can  produce  documentation  readable  by  the  other  teams/ communities. 

5.  I  can  clarify  expectations  to  the  other  team  s/contmuni  ties. 

6.  I  can  manage  deadlines  across  rime  zones, 

7.  I  am  able  to  develop  visual  aids  to  communicate  concepts  to  those  not  physically  present 

8.  1  feel  a  more  team-oriented  cooperative  spirit  across  the  teams. 

9.  1  have  a  better  appreciation  for  the  problems,  constraints  and  solutions  that  the  different  teams  encounter. 


Appendix  B.  JCUSI  Student  Survey  for  Unmanned  System's  Operational 
Site  Visit 


1 

1  would  rate  my  understanding  of  UAS  operations  as 

2 

This  trip  will  aid  my  understanding  of  UAS  operations  better  than  if  we  spent  this  lesson  in  the  classroom 

3 

1  would  rate  my  understanding  of  the  system  engineering  requirements  for  designing,  testing,  and 
maintaining  UAS  systems  as 

4 

I  would  rate  my  understanding  of  the  maintenance  and  logistics  requirements  for  UAS  systems  as 

5 

I  would  rate  my  understanding  of  the  global  nature  of  the  operational  US  UAS  systems  as; 

6 

I  would  rate  my  desire  to  become  a  UAS  operator  as 

7 

I  would  rate  my  desire  to  become  an  engineer  working  on  UAS  systems  as 

8 

I  would  rate  my  understanding  of  the  coordination  required  between  UAS  operators  and  soldiers  on  the 
ground  as 

9 

1  would  rate  my  understanding  of  the  need  for  joint  (Army,  Navy,  and  Air  Force)  operations  using  unmanned 
systems  as 

10 

I  would  rate  my  understanding  of  the  roll  of  unmanned  systems  in  military  operations 

11 

I  would  rate  my  understanding  of  possible  future  UAS  military  operations  as 

12 

1  would  rate  my  preparation  to  become  an  engineer  as 

13 

I  have  greater  knowledge  about  the  capabilities  and  limitations  of  current  UAS  systems 

14 

I  have  a  better  understanding  of  how  operational  needs  can  be  met  by  the  technical  capabilities  in  current 

UAS 

IB 

I  have  a  better  understanding  of  how  autonomous  systems  can  enhance  military  operations 

16 

I  have  a  better  appreciation  of  the  challenges  faced  by  field  operators  of  unmanned  systems 

NOTES:  The  ‘before  the  trip’  survey  questions  were  preceded  by  “Before  going  on  this  trip  ”  and  the  ‘after  the 
trip"  questions  by  ‘*After  going  on  this  trip  ”,  Questions  13-16  were  added  for  the  ‘after*  survey  to  measure  specific 
desired  learning  outcomes- 


Appendix  C-System  Requirements  Review  -  Expectations  and  Rubrics 

L  Purpose:  The  System  Requirements  Review  (SRR)  is  a  formal  briefing  by  your  project  team  to 
convince  your  mentor,  senior  reviewing  officer,  faculty  representative,  and,  as  appropriate,  your 
customer,  that  you  fully  understand  the  problem  you  are  trying  to  solve.  Recall  that  the  purpose  of  the 
SRR  is  to  ensure  that  you,  as  the  design  team,  your  mentor,  and  your  customer,  have  the  same 
understanding  of  the  requirements.  As  noted  in  DAU’s  System  Engineering  Fundamentals 7  (p.  104)  “The 
SRR  is  intended  to  confirm  that  the  user's  requirements  have  been  translated  into  system  specific 
technical  requirements  . . .  and  that  risks  are  well  understood  and  mitigation  plans  are  in  place.”  There 
should  be  NO  discussion  of  design  solutions  in  this  review. 

2.  SRR  Deliverables: 

Requirements:  30% 

o  Requirements  Traceability  Matrix  (MS  Excel)  Detailed  overview  in  an  Objective  Tree 
Functional  Description:  15% 

o  Functional  Flow  Block  Diagram  (FFBD)  of  the  top  level  system  functions  (MS  Visio) 

Can  be  presented  in  the  slides 
o  OV-1 

o  User  Interface  (UI)  Mockup 
Project  Plan:  20% 

o  Schedule  with  progress  to  date,  details  to  PDR  and  an  overview  of  the  entire  year.  (MS 
Project,  with  critical  path  analysis  in  slides) 
o  Risk  Analysis  (MS  PowerPoint  Slides) 
o  Configuration  Management  (MS  PowerPoint  Slides) 
o  Contemporary  Issues  (MS  PowerPoint  Slides) 

Presentation:  35%  (MS  PowerPoint) 
o  Communicate: 

•  Summary  of  your  project  requirements 

•  Current  status,  along  with  any  issues  and  your  plan  to  resolve  them 

•  Y our  detailed  plans  for  the  next  phase  of  the  project 

o  Present  a  dry  run  of  your  briefing  to  your  mentor  one  lesson  prior  to  the  SRR 
o  The  team’s  presentation  should  run  for  55  minutes  or  lass  to  allow  for  questions 
afterward 

3.  The  attached  rubrics  show  the  grading  weights  for  each  portion  of  each  deliverable.  NOTE:  If  any  of 
the  deliverables  submitted  for  this  design  review  earn  Unsatisfactory  (<67%)  marks,  they  must  be 
resubmitted  within  one  lesson  period  of  receiving  feedback.  If  the  presentation  receives  an  Unsatisfactory 
mark,  it  must  be  redone. 

4.  Each  individual  will  also  receive  a  grade  for  their  individual  performance  and  contributions  to  the 
team.  The  individual  grade  will  be  combined  with  the  team  grades  as  described  in  the  syllabus.  Each 
team  member  will  be  evaluated  based  on: 

The  appropriateness  of  work  accomplished  based  on  their  individual  skill  set 
The  amount  of  work  accomplished 
Inte  rperson  al  skill  s 

Bring  your  lab  notebooks  to  the  design  review  so  your  faculty  team  can  review  them.  Make  it  dear  in  the 
presentation  what  each  member  has  done. 

5.  Peer  Evaluations.  Prior  to  the  start  of  the  SRR  each  student  must  complete  a  Peer  Evaluation  using  the 
survey  posted  on  the  course  homepage. 


Requirements:  30% 
Requirements  Traceability  Matrix. 


Area  (weight)  1 

A  Work 

B  Work 

C  Work 

Unsatisfactory 

Require  merits 
Traceability  Matrix 
and  Objective  Tree 
(100%) 

*  Requirements  achievable, 
unambiguous,  consistent 

Sc  verifiable 

*  Cover  input/output, 
physical  constraints, 
performance  and 
environment 

*  Threshold  Sc  objective 
specified  if  applicable 

*  KPPs  identified 

*  Ambiguities  identified 
with  get- well  plans; 

»  Functional  allocation 
complete  and  logical 

*  Several  minor 
requirements 
issues 

»  Significant  problems 
with  requirements, 
problem  areas 
overlooked,  allocation 
problems 

*  Some  ambiguous 
requirements  lack  get- 
well  plans; 

•  No  objective 
requirements 

•  Some  acceptance  tests 
not  clear  or  specified 

*  No  KPPs 

■  Requirements 
poorly  written; 

■  Many  ambiguities, 
unclear  acceptance 
tests  or  allocation 
problems 

*  NOT  PRE¬ 
COORDINATED 
WITH 

CUSTOMER 

Functional  Description:  15% 
Functional  Flow  Block  Diagram 


Area  (weight) 

A  Work 

B  Work 

C  Work 

Unsatisfactory 

FFBD 

•  Thoroughly  describes  the 

*  Several  minor 

*  Some  functions 

*  Does  not  describe 

(60%) 

top  level  system 

oversights,  logic 

omitted  or  some  logic 

system  behavior; 

functions 

errors  or 

errors  describing 

*  Does  not  follow 

*  Logical  breakdown  and 

diagramming 

behavior 

FFBD  syntax  to  the 

arrangement  of  functions; 

■  Correct  syntax  lor  FFBD 

errors 

■  Significant  FFBD 
syntax  errors  which 
obscure  the  function 
being  described 

extent  that  the 
function  is  unclear 

OV- 1  and  U1 

*  Clearly  demonstrates 

*  Several  minor 

*  Simple  presentation 

•  Unclear 

Mockup 

proposed  system  concept 

ambiguities, 

*  Significant  portions  of 

operational 

(40%) 

*  Clearly  identifies 

oversights  or  other 

graphics  not  clear 

concept;  External 

boundaries  and 
relationships  to  external 
systems 

*  UI  shows  inputs  and 
outputs,  is  logical  and 
complete 

*  Prof es  sional/easy  to  read 

issues 

*  Significant  boundaries 
or  external 
relationships  not 
covered 

*  Significant  user 
interface  issues 

systems  and 
boundaries  not 
addressed 

*  User  interface 
provides  no  insight 
to  operator 

•  Sloppy 
presentation 

-  NOT  PRE¬ 
COORDINATED 
WITH 

CUSTOMER 

Project  Plan:  20% 

Includes:  Schedule  (MS  Project)  and  MS  PowerPoint  Slides  on  Rusk  Management,  Configuration  Management,  and 
Other  Considerations  (Environ./political/social,  Health/Safety,  Economic,  Manufacturability/ Sustain  ability,  Ethics) 


Area  (weight) 

A  Work 

B  Work 

C  Work 

Unsatisfactory 

Schedule:  (70%) 

*  Detailed  and  logically 
linked  set  of  tasks  that 
thoroughly  cover  the 
activities  required  to 
achieve  PDR 

*  Overview  of  entire 
project 

*  Includes  ri  sk 
management  Sc 

*  Flan  is  complete 
with  several 
minor  issues  with 
task  descriptions, 
linkage  or 
resource 

allocation 

*  Significant  tasks 
missing 

*  Some  tasks  vague  or 
not  linked 

*  Workload  allocated  to 
resources  but  has 
significant  balance 
problems 

*  Major  PDR  tasks 
missing 

•  Schedule  unusable 
because  not  linked, 
resourced,  or 
because  of  serious 
logic  problems 

documentation  tasks 
•  Resources  logically 
allocated  for  each  task 

Risk  Management 
(10%) 

*  Risks  are  assessed 
against  meeting  a 
documented  requirement 
and  are  logical; 

*  Solid  analysis  support  for 
probabilities  and 
consequences; 

*  Logical  and  achie  vable 
management  plans  and 
Strategies 

*  Few  general  risks; 

*  Few  probabilities 
and  consequences 
lack  solid  analysis 
support; 

*  Few  management 
plans  vague 

*  Some  generic  risks; 

*  Some  analysis  support 
for  probability  and 
consequence  analysis 
is  supported,  some  is 
vague;  some  is  missing 

*  Some  management 
plans  are  vague 

*  Mostly  generic  risks; 

■  Little  or  no  support 

for  probability  and 

consequence 

analysis; 

*  Some  key  risks  are 
ignored 

*  Management  plans 
either  do  not  exist  or 
are  vague 

Config  Mgt  (10%) 

■  Plans  workable,  cover  ail 
products,  evidence  of  use 

•  A  few  minor 
oversights 

*  Some  anticipated 
design  products  not 
discussed 

*  Only  cursory 
treatment  of 
configuration  control 

Contemporary 

Issues  (10%) 

*  Issues  thoughtfully 
considered; 

»  Concerns  and  actions 
required  documented  and 
in  the  project  schedule 

*  Few  minor  issues 

*  Minimal  thought  about 
issues.  Significant 
oversights 

*  Management  plans 
contain  little  detail 

*  Only  cursory 
attention,  with  vague 
if  any  management 
plans 

SRK  Presentation;  35% 

MS  PowerPoint  Slides  and  presentation  that: 

Clearly  communicate  the  purpose  and  goal  of  the  project  as  you  have  agreed  with  your  mentor 
Clearly  describe  the  current  status  of  the  project  including  get- we II  plans  for  any  issues 


C  ’  1  e;trlv  describe  your  plan  to  achieve  a  successful  FDR 


Area 

A  Work 

B  Work 

C  Work 

Unsatisfactory 

Clarity  of 

t 

Team  shows  a  clear 

*  Minor 

* 

Team  shows  basic 

* 

Requirements  not 

communication 

and  unified 

ambiguities 

understanding  of 

well  understood 

(60%) 

understanding  of  the 

•  Minor 

the  system 

or  major 

system  requirements 

detractions 

requirements  with  a 

inconsistencies  in 

* 

Concise 

from  briefing 

few  significant 

understanding 

communication 

disconnects 

amongst  the  team 

• 

Answers  to  questions 

* 

Communication  of 

* 

Communication 

are  concise  and 

requirements  is 

unclear  in  several 

accurate 

unclear  in  few  areas 

areas 

* 

Well  planned  and 

• 

A  few  questions  no! 

■ 

Several  questions 

executed 

well  answered 

not  answered 

* 

Team  has  a 

• 

Many 

well 

professional 

colloquialisms, 

* 

Briefing  lacks 

appearance  and  uses 

including,  “stuff 

planning 

intelligent  language 

like  this”,  “you 

» 

Briefing  more 

guys”,  and  “my 

than  10  minutes 

bad” 

over  rime 

Briefing  slides 

a 

Slides  support  the 

*  Several  minor 

* 

Information  on 

• 

Major  errors. 

(40%) 

delivery  of 

problems 

some  slides  does 

format  problems 

information 

not  support  the 

or  readability 

9 

Professional 

point  of  the  briefing 

issues 

a 

Readable 

* 

Several  format 

* 

Several  revisions 

• 

Well  organized 

inconsistencies  and 

after  slides 

* 

Adequately 

errors 

submitted  for 

referenced 

• 

Minor  changes  after 

review 

* 

Minor  if  any  issues 

slides  submitted  for 

* 

STUDENT 

review 

HOURS AND 

• 

Some  slides 

ACTIONS  NOT 

difficult  to  read 

REPORTED 

